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ABSTRACT 

The user population of the World Wide Web is increasing 
rapidly. People who have never had experience with computers are logging on, 
and, therefore, the novice user population is very large. These new users need 
to be trained in using this new technology. It is therefore important to 
investigate which training methods leave the users most satisfied. This paper 
presents the results of an experiment designed to assess user satisfaction 
with different training methods. Over 250 users took part in this experiment, 
measuring user satisfaction with eight different training methods: networked 
error training; conceptual error training; conceptual exploratory training; 
conceptual training; error training; error management training; exploratory 
training; and traditional training. There were not major differences in user 
satisfaction across training methods, but overall satisfaction scores were 
very high. This might mean that the users were in serious need of training and 
were satisfied to have received training, regardless of the type of training. 
(Contains 34 references.) (Author/MES) 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 



IR021338 ED 466 603 



r 



End-User Satisfaction in Training Novice Users to Surf the Web 



PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL HAS 
BEEN GRANTED BY 



G.H. -Marks 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 

1 



Dr. Jonathan Lazar 

Department of Computer and Information Sciences 
Towson University 
8000 York Road 
Towson, MD 2 1252, USA 
jlazar@towson.edu 

Dr. Anthony Norcio 
Department of Information Systems 
University of Maryland Baltimore County 
1 000 Hilltop Circle 
Baltimore, MD 21250, USA 
norcio@umbc.edu 



U.S. DEPARTMENT OF EDUCATION 
Office of Educational Research and Improvement 

EDUCATIONAL RESOURCES INFORMATION 
CENTER (ERIC) 

yi This document has been reproduced as 
received from the person or organization 
originating it. 

□ Minor changes have been made to 
improve reproduction quality. 



Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 



Abstract 

The user population of the world wide web is increasing rapidly. People who have never had experience with computers are logging on 
and therefore, the novice user population is very large. These new users need to be trained in using this new technology. It is therefore important 
to investigate which training methods leave the users most satisfied. This paper presents the results of an experiment designed to assess user 
satisfaction with different training methods. Over 250 users took part in this experiment, measuring user satisfaction with eight different training 
methods. There were not major differences in user satisfaction across training methods, but overall, satisfaction scores were very high. This might 
mean that the users were in serious need of training, and were satisfied to have received training, regardless of the type of training. 

Introduction 

As the popularity of the World Wide W eb grows, many people without previous computing experience are getting on the 
web (Lazar, 2001). The user population of the web includes people of all ages, gender, educational experience, computing 
experience, and disabilities (Shneiderman, 2000). Because the number of people who need to use the web is increasing, the number 
of people taking training classes related to the web is also increasing. Computer training is an important component of 
implementing a new technology. In some cases, professionals such as teachers are being required to take training classes in using 
the web (Wax, 2000). Newspaper articles are heralding the importance of computer training for using the World Wide Web 
(Mendels, 1999). In most cases, without appropriate training, users will not be able to effectively use the technology (Hoffer, 
George & Valacich, 1999; Lazar & Norcio, 2000a; Office of Technology Assessment, 1995; Whitten & Bentley, 1997). 

Since training in using the web has become necessary for a large number of people, it is important to research the 
effectiveness of different training methods for the web. Training can be presented in a number of different ways. For instance, 
training sessions can differ in how the material is presented to the users, what documentation is provided, or the amount of 
instructor participation. For instance, will the training be presented by telling users to type in certain commands, or will users be 
encouraged to explore? Will users be instructed on what errors could occur, and on how to respond to those errors? Which 
training methods will leave the user most satisfied? These are important research questions which this paper will address. 

Training Users in Responding to Errors 

In addition to the many considerations in a training session (such as room layout, lighting, time of day, documentation), 
there are a number of different approaches for presenting the training itself. For instance, users could be told exactly what to type 
in, or users could be encouraged to explore on their own. Training approaches can differ depending on whether they address the 
issue of errors, and if so, how they address those errors. Errors need to be addressed when training novice users, because errors 
occur frequently with novice users who are attempting a new comp uter task, and in the networked environment of the World Wide 
Web, there is an increased likelihood of error occurring (Greif & Keller, 1990; Lazar & Norcio, 1999b; Lazar & Norcio, 2000b; 
Lazonder & Meij, 1 995). Errors can frustrate novice users, who tend to blame themselves for errors (Arnold & Roe, 1987; Carroll & 
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Mack, 1984; Frese, Brodbeck, Zapf & Prumper, 1990; Lewis & Norman, 1986). In addition, novice users may spend a lot of time 
attempting to recover from an error sequence (Carroll & Carrithers, 1984). Even though errors are such a problem for novice users, 
unfortunately, most training methods for novice users do not address the errors that can occur (Frese & Altmann, 1 989). 

A number of methods for addressing errors in novice user training have been presented in the literature (Lazar & Norcio, 
1999a). Some of the different approaches to training users to respond to errors include error management training, exploratory 
training, and conceptual models. In error management training, errors are presented as opportunities for learning (Dormann & 
Frese, 1994; Frese & Altmann, 1989; Frese et al., 1991; Nordstrom, Wendland & Williams, 1998). In error management training, users 
are informed about the errors that may occur, and are instructed in strategies for coping with errors. Errors are presented as a 
positive step, rather than a negative setback. In exploration, users are given an overview of their environment (Greif & Keller, 1990; 
Wendel & Frese, 1987). Instead of being given step-by-step directions, users are taught how to navigate through their task 
environment. For instance, instead of being told to type in a specific command, such as Acd WWW = the user will be presented 
with an explanation of what the Acd= command is, and what options exist. The idea behind exploration is that users will have a 
better understanding of their environment, and therefore be better prepared to respond to the unexpected. Conceptual models are 
graphical or mathematical representations of a system that correspond closely to the real-world system (Santhanam & Sein, 1994). 
For instance, a conceptual model of the Internet might display how the usei^s computer communicates with numerous servers, to 
access the web page that the user has requested. Conceptual models assist users in understanding systems, and predicting the 
actions of systems. 

An important consideration related to the presentation of training is which training methods will leave the users with a 
high level of satisfaction. Since novice users are frustrated by errors, the expectation is that training methods that address errors 
should have higher levels of satisfaction, since the users will be better able to address errors, and therefore will become less 
frustrated. For instance, in two studies on error training in non-networked applications, subjects who received training in how to 
respond to errors reported lower levels of frustration than subjects who did not receive any training in responding to errors (Frese 
et al., 1991; Nordstrom et al., 1998). Users should leave a training session with a high level of satisfaction, confident that they will 
be able to tackle the computer tasks when they return to their standard (workplace or home) computing environment. 

We have created a research framework, which shows that by examining the training methods previously discussed in the 
research literature, some new hybrid methods of training have yet to be explored (Lazar & Norcio, 1999a). In addition, only one of 
these training methods (conceptual training) has been tested in the networked environment, where there is an increased 
opportunity for error. The resulting combinations of training methods can be seen in Figure 1. 

Research Methodology 

The research framework was used as the basis for the present experiment. Based on this research framework, eight 
different approaches for training novice users to use the web were tested. In total, 263 subjects (novice users) participated in the 
experiment. A novice user was defined as someone who has 1) not previously taken a class in using the Internet, and 2) does not 
use the Internet as a regular part of their job\ In addition, university students were excluded from the experiment, because they do 
not accurately represent the average novice user. The treatment groups in the experiment are displayed in Figure 1. The subjects 
were randomly assigned to treatment groups. Treatment group H is the control group, where subjects did not receive any 
assistance in responding to error. The average age of the subjects taking part in the experiment was 50.3 years old (std. dev. 9.9 
years) 



Treatment 

Group 


Conceptual 

Model 

(CM) 


Error 

Management 

(EM) 


Exploratory 

Training 

(ET) 


Training Methodology 
Name 


A 


Yes 


Yes 


Yes 


Networked Error Training 


B 


Yes 


Yes 


No 


Conceptual Error Training 


C 


Yes 


No 


Yes 


Conceptual Exploratory Training 


D 


Yes 


No 


No 


Conceptual Training 


E 


No 


Yes 


Yes 


Error Training 


F 


No 


Yes 


No 


Error Management Training 
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G 


No 


No 


Yes 


Exploratory Training 


H 


No 


No 


No 


Traditional Training 



Figure 1. Treatment Groups 



Each training session lasted four hours. The same protocol was followed in all of the training sessions: 

1. Subjects filled out human subjects forms, describing their rights in the experiment, as required by university policy and federal 

law. 

2. Subjects received a three hour training session. The training presented to them depended on which treatment group the training 

session belonged to. 

3. Subjects were given a list of 10 information gathering tasks on the World Wide Web, and subjects were given an hour in which 

to complete these tasks. The correct answers to these tasks could not be guessed, and there was only one correctanswer 
for each task. The specific tasks, as well as the results of the information gathering tasks, have been reported in (Lazar & 
Norcio, 2001). 

4. As soon as the subjects had completed the tasks, they completed the satisfaction questionnaire. 

In the experiment, data was collected on the task performance of the subjects, as well as the time performance of the 
subjects. In a previously published paper, it was reported that exploratory training was the most effective for improving task 
performance, and exploratory training and conceptual models were most effective for improving time performance (Lazar & Norcio, 
2001). Data on user satisfaction was collected as part of this experiment. The goal was to learn if there are any differences in user 
satis faction related to the type of training method presented. This paper reports on the findings of the satisfaction data. 

The satisfaction questionnaire was based on the Questionnaire for User Interaction Satisfaction (QU1S) (Norman, 
Shneiderman, Harper & Slaughter, 1998). The QUIS is a standard tool for evaluation of user satisfaction, which is used widely and 
has been previously validated in the literature (Chin, Diehl & Norman, 1988; Harper, Slaughter & Norman, 1997; Shneiderman, 1998; 
Slaughter, Harper & Norman, 1994). The QUIS is relatively long, and therefore only certain parts of the QUIS were used in the 
experiment. The different sections of the QUIS are designed to be used separately, as needed (Norman et al., 1 998). The part of the 
QUIS that was used in the study is Part 3 (Overall User Reactions). The other parts of the QUIS, such as questions about 
teleconferencing, technical manuals, and software installation, are not relevant to this study, and therefore were not included. The 
parts of the QUIS used in the satisfaction questionnaire are the dull-stimulating scale, the terrible -wonderful scale, the difficult-easy 
scale, the frustrating-satisfying scale, and the rigid -flexible scale. These scales represent overall user satisfaction with their 
experience. The actual satisfaction questionnaire used in the experiment is displayed in Appendix A. An earlier pilot study of 16 
users found that there were no problems in the clarity of the satisfaction questionnaire. 

Results 

The mean satisfaction scores of subjects are displayed in figure 2, organized by the training methods that the subject 
received. ANOVA tests were then performed to determine if there were any statistically significant differences (Maxwell & Delaney, 
1990). There are no statistically significant differences between any of the training groups on the dull-stimulating scale, the terrible - 
wonderful scale, or the difficult-easy scale. 



Figure 2. User responses on scales of satisfaction 



Group 


Terrible 


Frustrating 


Dull 


Difficult 


Rigid 


A 


7.9429 


7.1714 


7.8889 


6.4167 


8.0833 


B 


7.1905 


7.0000 


7.4762 


6.3182 


7.4762 


C 


7.3182 


6.2727 


7.5000 


6.1364 


8.0476 


D 


7.6000 


7.1522 


7.7045 


6.2045 


8.0930 


E 


7.5641 


7.2895 


7.7632 


6.9487 


8.2895 


F 


7.6522 


6.7826 


7.6522 


5.9565 


7.4783 
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G 


7.9143 


7.6857 


7.8824 


6.8000 


8.2857 


H 


7.9375 


7.6774 


8.1250 


7.0625 


8.3750 


Avg. 


7.6746 


7.1912 


7.7760 


6.5217 


8.0723 



Figure 2. User responses on scales of satisfaction 

On two satisfaction scales, the frustrating-satisfying scale and the rigid-flexible scale, statistically significant differences were 
found. A Duncan=s post-hoc test was run to determine where the statistically significant differences were. On the frustrating- 
satisfying scale, there is a significant difference between Conceptual Exploratory Training (Group C, with a mean of 6.2727) and 
Exploratory Training (Group G, with a mean of 7.6857) and Traditional Training (Group H, with a mean of 7.6774). On the rigid- 
flexible scale, there is a significant difference between Conceptual Error Training (Group B, with a mean of 7.4762) and Error 
Training (Group E, with a mean of 8.2895), Exploratory Training (Group G, with a mean of 8.2857) and Traditional Training (Group H, 
with a mean of 8.3750). There is also a significant difference between Error Management Training (Group F, with a mean of 7.4783) 
and Error Training (Group E, with a mean of 8.2895), Exploratory Training (Group G, with a mean of 8.2857) and Traditional Training 
(Group H, with a mean of 8.3750). 

Discussion 

In this experiment, subjects were asked to fill out 5 different scales of satisfaction from the QUIS, the dull-stimulating 
scale, frustrating-satisfying scale, terrible -wonderful scale, difficult-easy scale, and the rigid -flexible scale. There were only a small 
number of statistical differences in satisfaction between training groups, and these statistically significant differences appear only 
in two of the five scales of satisfaction. In the two scales where statistically significant differences did appear, the statistically 
significant difference is generally between the highest and lowest groups, not between one or two groups and ALL of the other 
groups. If there are truly significant differences in satisfaction, it is expected that it would appear on more scale s of satisfaction, or 
statistically significant differences would appear between more of the training groups. Therefore, it does not appear that there are 
any meaningful differences in the user satisfaction of the subjects related to training methods. It is important, however, to note that 
all of the satisfaction scores were very high. 

There are no major differences in satisfaction across the training methods of the experiment, but all satisfaction scores 
were very high. This is a very interesting finding. One of the requirements for subjects was that they had not previously taken a 
training class on using the Internet. It is possible that subjects had been staring at their computers, wondering about what the web 
is, and how they could use the Internet, for months before taking part in the training session. In this study, the training was offered 
to the employees at no cost. These employees might have been Astarved= for training, and might have simply been satisfied with 
having received training - any type of training - at no cost. Although the task and time performance varied greatly depending on 
the type of training received (Lazar & Norcio, 2001), the satisfaction level did not. Subjects might have been satisfied simply 
because they had received training, and now had more knowledge and felt more confident using the web. This concurs with a 
number of recent newspaper articles, which described users who were worried and uncomfortable using the web because they had 
not received training (Mendels, 1999; Wax, 2000). At this point of rapid growth in computer technology, the important factor in user 
satisfaction may not be related to the TYPE of training, but rather to having received training at all. 

Systems analysis and design and management information systems textbooks note that training is a very important factor 
in the user having a successful computing experience (Hoffer et al., 1999; Martin, DeHayes, Hoffer & Perkins, 1994; Whitten & 
Bentley, 1997). It is therefore distressing to note that in many cases, training may not be affordable or might be the first thing to be 
cut out of the budget, especially for schools and other non-profit organizations (Lazar & Norcio, 2000a; Mendels, 1 999; United 
States General Accounting Office, 1998). When non-profit organizations do spend money on training users, the training budgets 
are usually very low (Benton Foundation, 1997; Office of Technology Assessment, 1995). More community partnerships and 
unique approaches need to be developed to ensure that users receive the training that they need, to ensure that the technology 
gap between the Ahaves= and Ahave-nots= does not grow wider (Lazar & Norcio, 2000a). 

Summary 

An important finding of this study is that users want training. There were not major differences in the user satisfaction 
related to the different training methods. Rather, the novice users were very satisfied to received training at all, regardless of how 
the training was presented. This concurs with findings that without training, users are not successful in effectively utilizing 
information technology. Training needs to be made a priority in information technology budgets, to ensure that users can 
effectively use new technologies. 
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Appendix A. Satisfaction questionnaire 

Please circle the numbers which most appropriately reflect your impressions about this training experience. (Not Applicable = NA) 



terrible wonderful 

123456789 NA 



frustrating 


satisfying 


12 3 4 


5 6 7 8 9 NA 




dull 

12 3 4 


5 6 7 8 9NA 


stimulating 


difficult 


easy 




12 3 4 


5 6 7 8 9NA 




rigid 
12 3 4 


5 6 7 8 9 NA 


flexible 
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